Image Capture Technique

ABSTRACT

This disclosure relates to winking to capture image data using an image capture device that is associated with a head-mountable device (HMD). An illustrative method includes detecting a wink gesture at an HMD. The method also includes causing an image capture device to capture image data, in response to detecting the wink gesture at the HMD.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/563,238, filed on Jul. 31, 2012, and entitled “Image CaptureTechnique,” which claims the benefit of U.S. Provisional Application No.61/619,335, filed on Apr. 2, 2012, and entitled “Wink to Take a Photo onan HMD,” which are both herein incorporated by reference as if fully setforth in this description.

BACKGROUND

Computing devices such as personal computers, laptop computers, tabletcomputers, cellular phones, and countless types of Internet-capabledevices are increasingly prevalent in numerous aspects of modern life.Over time, the manner in which these devices are providing informationto users is becoming more intelligent, more efficient, more intuitive,and less obtrusive.

The trend toward miniaturization of computing hardware, peripherals,sensors, detectors, and image and audio processors, among othertechnologies, has helped open up a field sometimes referred to as“wearable computing.” In the area of image and visual processing andproduction, it has become possible to consider wearable displays thatplace a very small image display element close enough to one or both ofthe wearer's eyes such that the displayed image fills or nearly fillsthe field of view, and appears as a normal sized image, such as might bedisplayed on a traditional image display device. The relevant technologymay be referred to as “near-eye displays.”

Near-eye displays are fundamental components of wearable displays, alsosometimes called “head-mountable displays”. A head-mountable displayplaces a graphic display close to one or both of the wearer's eyes. Togenerate the images on the display, a computer processing system can beused.

Emerging and anticipated uses of wearable displays include applicationsin which users interact in real time with an augmented or virtualreality. These applications can be mission-critical or safety-criticalin some fields, such as public safety or aviation.

SUMMARY

This disclosure provides, in part, a method. The method includesdetecting a wink gesture at a head-mountable device (HMD). The methodalso includes causing an image capture device to capture image data, inresponse to detecting the wink gesture at the HMD.

This disclosure also provides, in part, a non-transitorycomputer-readable medium. The medium has stored therein instructionsthat, upon execution by a computing device, cause the computing deviceto perform functions. The functions include detecting a wink gesture atan HMD. The functions also include causing an image capture device tocapture image data, in response to detecting the wink gesture at theHMD.

This disclosure also provides, in part, a system of an HMD. The systemincludes an image capture device that is connected to the HMD. When theHMD is worn, the image capture device is configured to capture imagedata. The system also includes a wink-detection system that is connectedto the HMD. When the HMD is worn, the wink-detection system isconfigured to detect a wink gesture at the HMD. The system also includesa computer-readable medium. The medium has stored therein programinstructions that, upon execution by a computing device, cause thecomputing device to perform functions. The functions include causing theimage capture device to capture the image data, in response to thewink-detection system detecting the wink gesture at the HMD.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate an example of a wearable computing system.

FIG. 1C illustrates another example of a wearable computing system.

FIG. 1D illustrates another example a wearable computing system.

FIG. 2 illustrates an example of a wink-detection system.

FIG. 3 illustrates an example of a computing system.

FIG. 4 is a flow chart illustrating a method, according to anembodiment.

FIG. 5 illustrates the wink-detection system of FIG. 2 interacting withan eye area of an upward-looking user.

FIG. 6 illustrates the wink-detection system of FIG. 2 interacting withan eye area of a downward-looking user.

FIG. 7A illustrates the wink-detection system of FIG. 2 interacting withan eye area of a blinking user.

FIG. 7B illustrates the wink-detection system of FIG. 2 interacting withan eye area of a winking user.

DETAILED DESCRIPTION 1. Overview

This disclosure relates to winking to capture image data using a camera,such as a point-of-view camera, that is associated with a head-mountabledevice (HMD), such as a glasses-style wearable computer. An HMD caninclude a system, such as a proximity sensing system, that can detect awink by the HMD's wearer (when the HMD is being worn). Upon detecting awink, the HMD can responsively operate a camera to capture image data,such as, for example, an image or a frame of a video, among other typesof image data. To this end, the HMD can use its own camera (such as afront-facing camera) or can use a separate camera (such as a handheldcamera). These and other aspects of this disclosure are discussed inmore detail below in sections 2 and 3(a)-3(c).

Some HMDs can incorporate one or more other gestures into theimage-capturing process. As an example, another gesture, such as aninitial wink, a blink, a touchscreen gesture, or a head movement, canactivate a camera interface of an HMD. Once the camera interface isactive, a secondary gesture can trigger and/or operate one or more ofthe camera's features, such as a flash setting or zoom level. Then, awink can trigger the camera to capture image data. These and otheraspects of this disclosure are discussed in more detail below insections 3(d) and 3(e).

Also discussed in this disclosure are various other aspects of winkingto capture image data. For instance, in some HMDs, a time, place,particular user of the HMD, or another context in which the HMD operatescan activate, de-activate, or modify image-capturing functionality ofthe HMD. In addition, in some HMDs, a wink gesture can start or end aphotographic process, such as a time-lapse photography process. And someHMDs can use captured image data as input to one or more of the HMD'sfunctions, such as image recognition, mapping, and social media, amongothers. These and other aspects of this disclosure are discussed in moredetail below in sections 3(f)-3(h).

2. Device and System Architecture

a. Head-Mountable Devices

FIG. 1A illustrates an example of a wearable computing system 100. Thewearable computing system 100 includes a wink-detection system 136 andan image-capturing system 120. While FIG. 1A illustrates ahead-mountable device (HMD) 102 as an example of a wearable computingsystem, other types of wearable computing systems could be used. Asillustrated in FIG. 1A, the HMD 102 includes frame elements, includinglens frames 104, 106 and a center frame support 108, lens elements 110,112, and extending side arms 114, 116. The center frame support 108 andthe extending side arms 114, 116 are configured to secure the HMD 102 toa user's face via a user's nose and ears.

Each of the frame elements 104, 106, and 108 and the extending side arms114, 116 can be formed of a solid structure of plastic and/or metal, orcan be formed of a hollow structure of similar material so as to allowwiring and component interconnects to be internally routed through theHMD 102. Other materials can be used as well.

The lens elements 110, 112 can be formed of any material that cansuitably display a projected image or graphic. Each of the lens elements110, 112 can also be sufficiently transparent to allow a user to seethrough the lens element. Combining these two features of the lenselements can facilitate an augmented reality or heads-up display wherethe projected image or graphic is superimposed over a real-world view asperceived by the user through the lens elements.

The extending side arms 114, 116 can each be projections that extendaway from the lens frames 104, 106, respectively, and can be positionedbehind a user's ears to secure the HMD 102 to the user. The extendingside arms 114, 116 can further secure the HMD 102 to the user byextending around a rear portion of the user's head. The wearablecomputing system 100 can also or instead connect to or be affixed withina head-mountable helmet structure.

The HMD 102 can include an on-board computing system 118, a video camera120, a sensor 122, and a finger-operable touch pad 124. The on-boardcomputing system 118 is shown to be positioned on the extending side arm114 of the HMD 102. The on-board computing system 118 can be provided onother parts of the HMD 102 or can be positioned remote from the HMD 102.For example, the on-board computing system 118 could be wire- orwirelessly-connected to the HMD 102. The on-board computing system 118can include a processor and memory, for example. The on-board computingsystem 118 can be configured to receive and analyze data from the videocamera 120 and the finger-operable touch pad 124 (and possibly fromother sensory devices, user interfaces, or both) and generate images foroutput by the lens elements 110 and 112. The on-board computing systemcan take the form of the computing system 300, which is discussed belowin connection with FIG. 3.

With continued reference to FIG. 1A, the video camera 120 is shownpositioned on the extending side arm 114 of the HMD 102; however, thevideo camera 120 can be provided on other parts of the HMD 102. Thevideo camera 120 can be configured to capture image data at variousresolutions or at different frame rates. One or multiple video cameraswith a small form factor, such as those used in cell phones or webcams,for example, can be incorporated into the HMD 102.

Further, although FIG. 1A illustrates one video camera 120, more videocameras can be used, and each can be configured to capture the sameview, or to capture different views. For example, the video camera 120can be forward facing to capture at least a portion of the real-worldview perceived by the user. The image data captured by the video camera120 can then be used to generate an augmented reality where computergenerated images appear to interact with the real-world view perceivedby the user.

The sensor 122 is shown on the extending side arm 116 of the HMD 102;however, the sensor 122 can be positioned on other parts of the HMD 102.The sensor 122 can include one or more of a gyroscope, an accelerometer,or a proximity sensor, for example. Other sensing devices can beincluded within, or in addition to, the sensor 122 or other sensingfunctions can be performed by the sensor 122.

The finger-operable touch pad 124 is shown on the extending side arm 114of the HMD 102. However, the finger-operable touch pad 124 can bepositioned on other parts of the HMD 102. Also, more than onefinger-operable touch pad can be present on the HMD 102. Thefinger-operable touch pad 124 can be used by a user to input commands.The finger-operable touch pad 124 can sense at least one of a positionand a movement of a finger via capacitive sensing, resistance sensing,or a surface acoustic wave process, among other possibilities. Thefinger-operable touch pad 124 can be capable of sensing finger movementin a direction parallel or planar to the pad surface, in a directionnormal to the pad surface, or both, and can also be capable of sensing alevel of pressure applied to the pad surface. The finger-operable touchpad 124 can be formed of one or more translucent or transparentinsulating layers and one or more translucent or transparent conductinglayers. Edges of the finger-operable touch pad 124 can be formed to havea raised, indented, or roughened surface, so as to provide tactilefeedback to a user when the user's finger reaches the edge, or otherarea, of the finger-operable touch pad 124. If more than onefinger-operable touch pad is present, each finger-operable touch pad canbe operated independently, and can provide a different function.

FIG. 1B illustrates an alternate view of the wearable computing system100 illustrated in FIG. 1A. As shown in FIG. 1B, the lens elements 110,112 can act as display elements. The HMD 102 can include a firstprojector 128 coupled to an inside surface of the extending side arm 116and configured to project a display 130 onto an inside surface of thelens element 112. A second projector 132 can be coupled to an insidesurface of the extending side arm 114 and can be configured to project adisplay 134 onto an inside surface of the lens element 110.

The lens elements 110, 112 can act as a combiner in a light projectionsystem and can include a coating that reflects the light projected ontothem from the projectors 128, 132. In some embodiments, a reflectivecoating may not be used (such as, for example, when the projectors 128,132 are scanning laser devices).

In some embodiments, other types of display elements can also be used.For example, the lens elements 110, 112 themselves can include one ormore transparent or semi-transparent matrix displays (such as anelectroluminescent display or a liquid crystal display), one or morewaveguides for delivering an image to the user's eyes, or one or moreother optical elements capable of delivering an in focus near-to-eyeimage to the user. A corresponding display driver can be disposed withinthe frame elements 104, 106 for driving such a matrix display.Alternatively or additionally, a laser or LED source and scanning systemcould be used to draw a raster display directly onto the retina of oneor more of the user's eyes.

The wink-detection system 136 is shown in FIG. 1B as a proximity-sensingsystem including a light source 138 and a light sensor 140 affixed tothe extending side arm 114 of the HMD 102. Although the wink-detectionsystem 136 is shown as a proximity-sensing system, other types ofwink-detection systems can be used. As discussed below in connectionwith FIG. 2, a wink-detection system can also include other numbers oflight sources (including no light sources) and can include elementsother than those shown in the wink-detection system 136. Additionally,the wink-detection system can be arranged in other ways. For example,the light source 138 can be mounted separately from the light sensor140. As another example, the wink-detection system 136 can be mounted toother frame elements of the HMD 102, such as, for example, to the lensframes 104 or 106, to the center frame support 108, or to the extendingside arm 116.

FIG. 1C illustrates another example of a wearable computing system 150.The wearable computing system 150 includes an image-capturing system156. The wearable computing system 150 can be coupled to awink-detection system, although a wink-detection is not shown in FIG.1C. While FIG. 1C illustrates an HMD 152 as an example of a wearablecomputing system, other types of wearable computing systems could beused. The HMD 152 can include frame elements and side arms such as thosediscussed above in connection with FIGS. 1A and 1B. The HMD 152 can alsoinclude an on-board computing system 154 and a video camera 156, such asthose discussed above in connection with FIGS. 1A and 1B. The videocamera 156 is shown to be mounted on a frame of the HMD 152; however,the video camera 156 can be mounted at other positions as well.

As shown in FIG. 1C, the HMD 152 can include a single display 158, whichcan be coupled to the HMD. The display 158 can be formed on one of thelens elements of the HMD 152, such as a lens element having aconfiguration as discussed above in connection with FIGS. 1A and 1B. Thedisplay 158 can be configured to overlay computer-generated graphics inthe user's view of the physical world. The display 158 is shown to beprovided in a center of a lens of the HMD 152; however, the display 158can be provided in other positions. The display 158 is controllable viathe computing system 154, which is coupled to the display 158 via anoptical waveguide 160.

FIG. 1D illustrates another example of a wearable computing system 170.The wearable computing system 170 can include an image-capturing system178 and a wink-detection system (not shown in FIG. 1D). The wearablecomputing system 170 is shown in the form of an HMD 172; however, thewearable computing system 170 can take other forms as well. The HMD 172can include side arms 173, a center frame support 174, and a bridgeportion with a nosepiece 175. In the example shown in FIG. 1D, thecenter frame support 174 connects the side arms 173. The HMD 172 doesnot include lens-frames containing lens elements. The HMD 172 can alsoinclude an on-board computing system 176 and a video camera 178, such asthose discussed above in connection with FIGS. 1A and 1B.

The HMD 172 can include a single lens element 180, which can be coupledto one of the side arms 173 or to the center frame support 174. The lenselement 180 can include a display, such as the display discussed abovein connection with FIGS. 1A and 1B. The lens element 180 can beconfigured to overlay computer-generated graphics upon the user's viewof the physical world. In an example, the single lens element 180 can becoupled to the inner side (the side exposed to a portion of a user'shead when worn by the user) of the extending side arm 173. The singlelens element 180 can be positioned in front of or proximate to a user'seye when the user wears the HMD 172. For example, the single lenselement 180 can be positioned below the center frame support 174, asshown in FIG. 1D.

b. Proximity-Sensing Wink-Detection System

FIG. 2 illustrates an example of a wink-detection system 200 interactingwith an eye area 204. The eye area 204 can include the eye surface,eyelids, and portions of the face around the eye. The wink-detectionsystem 200 includes two light sources 202A and 202B that are configuredto provide light (light shown as dashed lines) to the eye area 204, anda light sensor 206 that is configured to detect reflected light (alsoshown as dashed lines) from the eye area 204. The wink-detection systemcan further include a processing unit (not shown in FIG. 2) that canperform computing functions. In particular, the processing unit candrive the light sources 202A-B, receive readings from the light sensor206, process the readings to determine aspects of the eye area 204, orperform combinations of these functions, among other functions.

i. Light Source

The wink-detection system 200 is shown to use two light sources 202A-Bto provide light to the eye area 204. While two light sources are shown,in general, a wink-detection system can use any suitable number of lightsources to illuminate the eye area. Further, some wink-detection systemsinclude no light sources. Instead, these systems can detect ambientlight or other illumination coming from the eye area.

In systems using light sources, the light sources can be any type oflight source. For example, the light sources can be light-emittingdiodes (LEDs), laser diodes, incandescent sources, gas dischargesources, or combinations of these light sources, among other types oflight sources. The light sources can be integrated with the system orexternally connected to the system, and can be driven by a light sensoror a processing unit. The light sources can emit light of any suitablefrequency or intensity. In an embodiment, the emitted light can have anintensity that is in a range that is safe for the user's eye. And thelight can have a frequency that renders the light invisible to humans inorder to avoid irritating the user. To this end, the light can beinfrared light, near-infrared light, or the like. Note that somewink-detection systems can use visible light or high-intensity light,depending on the desired configuration of the wink-detection system.

In some embodiments, the light sources can be aimed at specific portionsof the eye area. For example, the light sources 202A-B are aimed at anupper portion and a lower portion of the eye, respectively, near theinside corner 208 of the eye. In other cases, a single light source canbe directed at the whole eye area or at a part of the eye area, such as,for example, at one eyelid or at the center of the eye. As anotherexample, several light sources can each aim at respective various pointson the eye area, illuminating the eye at each of the various points.Light sources can also differ in the amount of the eye area to whichthey provide light (termed a spot size). For example, one light sourcecan have a spot size that provides light to the entire eye area, andanother light source can focus on a relatively small point on the eye.Further, the shape of the illuminated area can influence the behavior ofthe system. For example, if a light source illuminates a narrowhorizontal area across the top of the eye area, the amount of reflectedlight can depend on whether the upper eyelid covers that particularheight. As another example, a light source that provides light to theentire eye area can allow a system to detect the difference between acompletely closed eye and an eye that is almost completely closed.

In addition, a light source can use modulated or pulsed light todistinguish that light source from other light sources and from ambientlight. In particular, each light source can be configured to pulse at aparticular pattern so that the sensor can determine which light sourcesent the light based on the on/off pattern of the light. Because ambientlight may not follow any such pattern, the light from the system's lightsources can be distinguished from ambient-light noise by processing themeasured light signal. Note that other light characteristics can be usedto distinguish between light sources and/or ambient light. Examples ofsuch light characteristics include frequency (color) and intensity ofthe light.

In some implementations, in an HMD that uses a light source, the lightsource can include a structured light scanner. The structured lightscanner can be configured both to project light onto one or moresurfaces, and to detect the light projection at the one or moresurfaces. Of course, in some implementations, the structured lightscanner can perform one of these functions, and another device or set ofdevices can perform the other function. When the HMD is worn, thestructured light scanner can be aimed at a wearer's eye area.Accordingly, the structured light scanner can project light onto part orall of the eye area. In addition, the structured light scanner candetect the projected light, and based on the deformation of the detectedlight relative to the projected light, for example, the scanner cancalculate information related to the shape of part or all of the eyearea. The information can be calculated on a real-time basis.Accordingly, as the wearer's eye shape changes, the real-timeinformation can be used to detect eye gestures.

The HMD need not include a structured light scanner for carrying outstructured light scanning; instead, the HMD can include another deviceor set of devices configured to carry out structured light scanning,whether that device or set of devices is known or has yet to bedeveloped. In addition, the structured light scanning can be performedwith respect to light that is not visible to the human eye (such as, forexample, infrared light) or with respect to light that is visible to thehuman eye. In addition, an HMD can include multiple light scanners, forexample, to scan areas at and around both of the wearer's eyes. In adifferent configuration, an HMD can include a single light scanner thatis configured to scan areas at and around both of the wearer's eyes.

Further, the light sources can include elements that allow the system todynamically change the generated light's frequency, intensity, spotsize, shape, focus, or combinations of these properties, among othertypes of properties. In addition, the light sources can couple with oneor more mechanical actuators or servos to facilitate changing the lightsource's position, light direction, or both. In this way, the system canallow for dynamic calibration and adjustments of the light sources.

ii. Light Sensor

The wink-detection system 200 also includes a light sensor 206 that isconfigured to detect light reflected from the eye area 204. As used inthis disclosure, the term “reflected” can refer to a variety ofinteractions between light and an eye area, including those interactionsthat direct the light toward a light sensor. Examples of suchinteractions include mirror reflection, diffuse reflection, andrefraction, among other scattering processes. The sensor can be any typeof light-sensitive element or device that is capable of outputting ameasurable change in response to changes in light intensity. Forinstance, the sensor can be a photodiode, an electro-optical sensor, afiber-optic sensor, or a photo-detector, among other examples. Further,the sensor can be configured to detect a specified frequency of light ora specified range of frequencies. In some implementations, thesensitivity of the sensor can be designed for specified frequencies andintensities of light.

The sensor can be positioned to detect light reflected from particularportions of the eye area. For example, the sensor can be positionedabove the eye to detect light reflecting from the top of the eye whenthe eye is open, and from the upper eyelid when the eye is closed. Inthis way, the sensor can detect the amount of the eye that the uppereyelid covers. In some embodiments, the light sensor can be aligned atan oblique angle with respect to the eye area (for example, according tothe configuration of the sensor 140 shown in FIG. 1B). In otherarrangements, the sensor can point directly at the eye area and can beaimed toward the center of the eye area.

In some arrangements, the system can detect light reflected from asecond eye area. For example, the system can receive light data fromanother light sensor, which can detect light from a user's other eyearea. Alternatively, one light sensor can be positioned to detect lightfrom both eye areas.

In addition, the system can adjust and calibrate the behavior of thesensor, for example, by changing the sensor's position, direction,frequency response, sensitivity, detectable area size or shape, orcombinations of these, among others. This can be performed based on thecontext in which the system is used—for example, whether the system iscalibrated to a particular user, an intensity of ambient light, thelight sources used, a battery level of the device, or the like. Forexample, the sensor can be coupled to mechanical actuators for changingits position and direction. As another example, the sensor can includechangeable filters and baffles for filtering out different frequenciesof light.

A sensor that detects light from multiple sources can differentiatebetween the signals from each light source. For example, if the systemuses a different pulsing pattern for each light source, then the sensorcan separate signals based on the detected pulsing characteristics ofdetected light. Additionally, the light sources can alternate when theyilluminate the eye area. In such an arrangement, the sensor canassociate a measurement of light with a source based on which source wason at the time that the light was measured. If the light sourcesilluminate different sections of the eye area, then the separate signalscan be further associated with the respective eye-area portions. Inother arrangements, the sensor can measure a single light intensitybased on light from all the sources, without differentiating between thesources.

c. Other Wink-Detection Systems

Other wink-detection systems can include one or more cameras configuredto capture video or still images of an eye area. Based on the capturedvideo or still images, a system can recognize movements of the eye andeye area and, in particular, can determine wink gestures.

Other wink-detection systems can use mechanical sensors to detect themotion of a user's eyelids and, from the detected motion, determine thatthe user is winking. As an example, a wink-detection system can beequipped with an electromyogram or a similar device that is configuredto evaluate electrical activity that is produced by skeletal muscles atthe wearer's eye area of interest; such a device can be used, inessence, to “hear” movements of muscles at the eye area. As anotherexample, the wink-detection system can be equipped with a vibrationdetector that is configured to detect relatively subtle vibrations atthe wearer's eye area of interest. This disclosure is not limited to thewink-detection systems discussed above; this disclosure contemplates anywink-detection system that is known or has yet to be developed.

In addition, although the wink-detection systems are discussed above inthe context of detecting wink gestures, each of the wink-detectionsystems discussed above can be configured more generally to function asan eye-gesture detection system that is configured to detect not onlywink gestures, but also other eye gestures, such as a squint or a blink.

d. Processing and Other Elements

The processing unit in the wink-detection system can be ageneral-purpose processor, a specialized processor, or both. Theprocessor can be integrated with the light sensor or sources, or theprocessor can connect to the light sensor and sources through a bus ornetwork connection. Further, the processor can include or connect to anon-transitory computer-readable medium, such as a hard disk, a memorycore, a memory drive, a server system, or a combination of these, amongothers. The computer-readable medium can store at least the programinstructions for directing the processor to execute the functionsassociated with any method provided in this disclosure.

The wink-detection system can include various other elements including,for instance, additional processing, sensing, lighting, or interfaceelements. Some wink-detection systems can include a motion sensor (agyroscope or an accelerometer, for example) to detect when the systemmoves. This can enable the system, for example, to determine whether achange in detected light could be due to a movement of the light sensor,with respect to the eye area, as opposed to a movement of the eyes oreyelids.

In some implementations, the wink-detection system can be integrated inor with a computing system, such as the wearable computing systemsdiscussed above in connection with FIGS. 1A-1D. In theseimplementations, the wearable computing systems can help a user tointerface with the wink-detection system, for instance, to specify userpreferences, change system settings, perform calibration processes, orperform any combination of these functions, among other functions.

FIG. 3 illustrates an example of a computing system 300. The computingsystem 300 can include at least one processor 302 and system memory 304.In an implementation, the computing system 300 can include a system bus306 that communicatively connects the processor 302 and the systemmemory 304, as well as other components of the computing system 300.Depending on the desired configuration, the processor 302 can be anytype of processor, such as, for example, a microprocessor (μP), amicrocontroller (μC), a digital signal processor (DSP), or anycombination of these, among others. Furthermore, the system memory 304can be of any type of memory now known or later developed including butnot limited to volatile memory (such as RAM), non-volatile memory (suchas ROM, flash memory, or the like) or any combination of these.

The computing system 300 can include various other components as well.For example, the computing system 300 includes an A/V processing unit308 for controlling the graphical display 310 and the speaker 312 (viathe A/V port 314), one or more communication interfaces 316 forconnecting to other computing devices 318, and a power supply 320. Thegraphical display 310 can be arranged to provide a visual depiction ofvarious input regions provided by the user interface 322. Note, also,that the user interface 322 can be compatible with one or moreadditional user-interface devices 328 as well.

Furthermore, the computing system 300 can also include one or more datastorage devices 324, which can be removable storage devices,non-removable storage devices, or a combination of these. Examples ofremovable storage devices and non-removable storage devices includemagnetic disk devices such as flexible disk drives and hard-disk drives(HDD), optical disk drives such as compact disk (CD) drives or digitalversatile disk (DVD) drives, solid state drives (SSD), or a combinationof these, among any other storage device now known or later developed.Computer storage media can include volatile and nonvolatile, removableand non-removable media.

The computing system 300 can communicate using a communication link 316(a wired or wireless connection) to a remote device 318. The remotedevice 318 can be any type of computing device or transmitter includinga laptop computer, a mobile telephone, or tablet computing device, orthe like, that can be configured to transmit data to the computingsystem 300. The remote device 318 and the computing system 300 cancontain hardware to enable the communication link 316, such asprocessors, transmitters, receivers, antennas, or the like.

In FIG. 3, the communication link 316 is illustrated as a wirelessconnection; however, wired connections can also be used. For example,the communication link 316 can be a wired serial bus such as a universalserial bus or a parallel bus, among other connections. The communicationlink 316 can also be a wireless connection using, for example,Bluetooth® radio technology, communication protocols described in IEEE802.11 (including any IEEE 802.11 revisions), Cellular technology (suchas GSM, CDMA, UMTS, EV-DO, WiMAX, or LTE), or Zigbee® technology, amongother possibilities. The wired or wireless connection can be aproprietary connection as well. The remote device 330 can be accessiblevia the Internet and can include a computing cluster associated with aparticular web service such as, for example, social networking, photosharing, or address book.

3. Operation

FIG. 4 is a flow chart illustrating a method, according to someembodiments. At block 402, the method 400 involves detecting a winkgesture at an HMD. At block 404, the method 400 involves causing animage capture device to capture image data, in response to detecting thewink gesture at the HMD.

a. Detecting a Wink Gesture

As mentioned above, at block 402, the method 400 involves detecting awink gesture at an HMD. A wink gesture can be detected in various ways.For instance, a camera-based system can capture video or still imagesthat show both of a user's eyes. Then, the system can recognize a winkgesture when one of the user's eyes closes. As another example, acamera-based system can capture images of one of the user's eyes andrecognize a wink gesture from characteristics of this eye's movement.For instance, a winking eye can tend to close slower than a blinkingeye, or the eyelid and skin around a winking eye can be more wrinkled orstrained than the corresponding eye area of a user that is closing botheyes.

Another wink detection system can measure the physical movements of auser's eyelids by physically interacting with a user's eyelids. Forexample, a system can track the movements of mechanical actuators thatare connected to a user's eyelids. As another example, a system canwirelessly receive motion data from motion sensors affixed to the user'seyelids. In either case, or in similar systems, the system can detectthe closure of a single eye and responsively determine that the eye iswinking.

The following discussion describes functions of a proximity-sensingsystem in the process of detecting a wink. The following descriptionmerely serves as an example and should not be construed to indicate thata proximity-sensing technique is more favorable than anotherwink-detection technique.

i. Providing Light to an Eye Area

As discussed above, a wink-detection system can include one or morelight sources. These light sources can be controlled by a light sensoror a processing unit. When in use, the light sources can provide lightto portions of an eye area. The eye area can include the user's eyesurface, eyelids, and portions of the face around the eye. The lightsources can provide light to some or all of the eye area.

The method 400 can involve the system providing light to the eye area byway of one or more light sources. The light sources can constantlyprovide light to portions of the eye, or they can provide light to theeye intermittently. For example, the sources can alternate being on andoff to facilitate distinguishing between the signals from each lightsource. Further, the on/off characteristics can help a sensor todifferentiate between ambient light and artificial light signals. Insome embodiments, a system can include both always-on and intermittentlight sources.

Because facial structures can vary on a user-by-user basis, some systemscan calibrate the direction, position, and spot size/shapecharacteristics of the light sources based on detected facialcharacteristics. For example, a system can determine the direction fromthe light sources to the center of an eye area using, for example, gazetracking, glint detection, or video recognition. Then, the system canmodify the arrangement of light sources so that at least one lightsource is aimed at the area around the center of the eye area.

ii. Receiving Light Data from a Light Sensor

The method 400 can involve the system receiving light data from a lightsensor. The light data indicates at least one characteristic of lightreflected from the eye area. The sensor can be configured to detectcertain aspects of the light, such as frequency and intensity of thelight. The sensor can detect other aspects of the detected light, suchas polarization, coherence, phase, spectral width, modulation, orcombinations of these aspects, among other aspects.

The light sensor can be arranged to detect light reflected from aparticular portion of the eye area or to detect light from the entireeye area. Additionally, the sensor can be specially designed to detectlight with certain attributes, such as, for example, a certain frequencyof modulation, a frequency of light, light with a particularpolarization, or combinations of these attributes, among otherattributes.

Further, the system can calibrate and adjust the characteristics of thesensor. For example, if the sensor is used with near-IR light sources,the sensor can be configured to filter out light that is not in thenear-IR frequency range to avoid a noisy signal. As another example, ifa wink-detection system is mounted high above the eye area, the systemcan detect the position of the eye and responsively aim the sensor lowerto capture the eye area. As another example, in response to detectingthat the light sources are not as bright as they were previously, thesystem can increase the sensitivity of the sensor to compensate for thelower light intensity.

The light data from the sensor can be received as discretelight-intensity measurements over time. Also, light data can representone combined signal from all light sources and eye-area portions, or thedata can include multiple data sets with each data set representing aparticular light source or detected portion of the eye area.

The intensity of light detected from a portion of the eye can changebased on the characteristics of the eye at the specified point. Inparticular, a sensor can detect more light when aimed at the skinsurrounding the eye (including the eyelids) than it detects when aimedat the surface (the sclera, cornea, or the like) of the eye, because of,among other considerations, the different light-scatteringcharacteristics of human skin and eye surface. Therefore, an increase indetected light from a particular portion of the eye area can beindicative of an eye movement that increases the amount of skin thatoccupies the portion of the eye area from which the sensor is detectinglight. For example, a sensor that detects light from the surface of aneye when the eye is open (relatively less light) can also detect lightfrom the eyelid when the eye is closed (relatively more light).

In addition to representing an eye closing movement, an increase inlight intensity detected by the sensor can represent other eyemovements. For example, FIG. 5 shows the detection system 200 of FIG. 2interacting with an eye area 500, in which the eye is looking up. Asshown, the bottom eyelid 504 has moved up into the path of the lightprovided by the source 202B. The intensity of the light detected by thesensor 206 and provided by the light source 202B, therefore, canincrease as a result of the eye movement, because more skin would beilluminated by this source than in the situation depicted in FIG. 2.Meanwhile, the light provided by the source 202A still illuminates thetop of the eye, without illuminating the eyelid 502 as it does in thesituation of FIG. 2. Hence, the intensity of light detected from thesource 202B can remain unchanged, and the overall detected lightintensity from both sources can therefore increase as a result of theeye movement.

As another example, FIG. 6 shows the detection system 200 interactingwith an eye area 600, in a scenario in which the eye is looking down. Asshown, the user's top eyelid 602 has moved down and into the path of thelight provided by the source 202A. The intensity of the light detectedby the sensor 206 from the light source 202A, therefore, can increase asa result of the eye movement, because more skin can be detected than inthe situation depicted in FIG. 2. Meanwhile, the light from the source202B still does not illuminate the top eyelid 602. Hence, the intensityof light detected from the source 202B would remain unchanged, and theoverall detected light intensity from both sources can increase as aresult of the eye movement.

iii. Detecting Data Indicating a Wink Gesture

The method 400 can involve detecting data indicating a wink gesture,based on the received light data. The light-scattering characteristicsof the skin and eye surface are such that when the eye closes, the lightdetected by the wink-detection system can increase due to an increase inthe area of skin that reflects light (or as a result of a decrease inthe area of the eye that reflects light). Therefore, an increase inlight can be the result of a wink gesture.

Additionally, the characteristics of a light increase can indicatewhether the corresponding eye movement is a wink or some other movement.For example, the amount of the increase can indicate whether the eyesare partially closed (as in a squint) or fully closed (as in a wink). Asanother example, the movement of closing a single eye (wink) can beslower than the movement of closing both eyes (blink).

More particularly, the increase in light that results from a blinkgesture can be smaller than the increase in light that results from awink gesture. For example, in a wink gesture, the eyelids and skinaround the eye can wrinkle more than in a blink gesture. The resultingwrinkles can reflect more light to the sensor than the flat skinassociated with a blink gesture would reflect. To illustrate, FIGS. 7Aand 7B show the wink detection system 200 interacting with blinking(FIG. 7A) and winking (FIG. 7B) eyes. As shown, the blinking eyes 702and 704 close flatly, so that the light spots 710 and 712 illuminateflat eyelid skin on the eye 702. In contrast, the eyes 706 and 708 areinvolved in a winking gesture. Due to the winking gesture, the eyelidand skin near the eye 706 is flexed and wrinkled (wrinkles shown assolid lines). Therefore, the same illuminated spots 710 and 712encounter folded and stressed skin on the eye 706. Hence, the lightreflected from the winking eye 706 can be different from the lightreflected by the blinking eye 702.

To distinguish a wink from other eye movements, the wink-detectionsystem can store data indicating the amount of light that reflects to asensor as a result of a wink gesture, and data indicating the lightintensity that results from other eye movements (such as a blinkgesture, a squint gesture, or a change in gaze direction). Then, when aneye movement is detected, the system can compare the current lightintensity to the stored data indicating the relative light intensitiesto determine whether or not the eye movement is the result of a winkgesture. The stored data can indicate the maximum or average amplitudesof light intensity associated with each eye movement. In some cases, thedata can also indicate the time-based changes in light intensity thatresult from various eye movements. For example, because an eye can closeslower in a wink than in a blink, the stored data can indicate acorresponding slower change in detected light intensity resulting from awink gesture than from a blink gesture. Further, the system can use theduration of a wink, the eye-opening speed after the closure, changes inintensity while the eye is closed, or the like as bases for determinethat a particular change in light indicates a wink gesture.

Depending on the portions of the eye that are illuminated and measured,a wink can be detected in different ways. For example, in the system200, the light from the top and bottom eyelids can be separatelymeasured, and increases in detected light can be recognized for eacheyelid. In other arrangements, the movement of a single eyelid can betracked, or the overall eye area can be measured.

In some implementations, a system can measure light from both of auser's eyes to confirm a detected wink gesture. In particular, once asystem has detected light data from a first eye area that indicates awink gesture, the system can compare these data to light data from thesecond eye area. By comparing the data from each eye area, the systemcan verify that a given action of the user's eyes corresponds to a winkgesture.

b. Determining a Gaze Direction

In addition to detecting a wink gesture, the system can determine a gazedirection, which can represent the direction along which the eye isoriented while winking (and before or after the wink). In particular,the method 400 can involve determining a gaze direction, based on thedetected wink gesture. The system can determine the gaze direction basedon characteristics of the eye area before, during, or after a detectedwink gesture.

In some implementations, the system can determine the gaze direction bycollecting and analyzing data other than the data used to detect thewink gesture. For example, a system can use a gaze tracker to track themovement of a user's pupil, glint, or other gaze-directioncharacteristics. Then, when the system detects a wink gesture, thesystem can use the last tracked position of the user's eye to determinethe gaze direction. In particular, the system can store datarepresenting a central position of the pupil (when the user is lookingstraight forward). Then, the system can estimate the difference betweenthe central position and the detected position of the pupil andcalculate a corresponding angular difference between the centralposition and the detected position. The calculated angle can thereforerepresent the difference between a “looking straight forward” gazedirection and the detected gaze position.

In some implementations, the gaze direction can be determined based onthe same data that the system uses to detect the wink gesture. Forexample, in a camera-based system, the system can be programmed torecognize the eye position from the captured video or still images thatare also used to recognize the winking motion. In particular, the systemcan track the movements of the user's pupil, with respect to other partsof the user's face, before the wink gesture. Then, when a wink gestureis detected, the system can refer to the tracked movement data todetermine the position of the pupil before the wink gesture. Similarly,the system can determine the gaze direction from the position of theuser's pupil immediately following the blink gesture.

The wink-detection system using proximity sensing can also determine thegaze direction from much or all of the same data as was used to detectthe wink gesture. In particular, as shown in the situations depicted inFIGS. 2, 5, and 6, the characteristics of the detected light can changebased on the direction along which the eye is oriented before and afterwinking. For example, the light detected by the system 200 can increaseas a result of an eye looking either up (as in FIG. 5) or down (as inFIG. 6). Hence, if the system 200 is configured to differentiate betweenthe signals from the source 202A and the signals from the source 202B,the increases in light intensity from each source can be associated withcorresponding eye movements.

To facilitate associating light-intensity data with eye-movementinformation, the system can collect and store representativelight-intensity data for known eye movements. For example, the systemcan be programmed with characteristic light-intensity levels thatcorrespond with a particular gaze direction. Alternatively,user-specific data can be gathered. For instance, a user can beinstructed to follow a calibration procedure to store particularintensity data associated with the particular user's facialcharacteristics. In particular, the system can prompt the user to lookin different directions, such as, for example, by using audio or textcommands, or by displaying an indicator in the direction that the usershould be looking. Then, the system can store the intensity of lightthat is detected from the user's eye area while the user is looking inthe different directions.

Further, the system can adjust the representative light-intensity levelsto better match the associated gaze directions. In particular, if thesystem determines that a representative level does not correctlyrepresent the light that can be detected when the eye is looking in theassociated gaze direction, then the system can responsively adjust therepresentative level to a level that does represent the light that canbe detected when the eye is looking in the gaze direction. For example,if the system detects that the most common detected light-intensitylevel (likely associated with a user looking straight ahead) is muchlower than the recorded intensity level associated with the straightahead gaze direction, the system can responsively lower therepresentative level to match the previous readings.

In addition, the system can calibrate the stored list of light-intensitylevels for a particular context in which the method is used. Forexample, a system that is used by multiple users can storerepresentative light-intensity levels for each user. When the userchanges, the system can responsively change the list of levels that ituses.

The system can then compare light-intensity levels before and/or afterthe wink gesture to the characteristic or recorded readings. By matchingthe detected intensity level(s) to representative levels, the system candetermine a possible gaze direction at the time of the wink.

Additionally, the system can store characteristic or user-specificlight-intensity data related to gaze directions with an eye in a closedstate (for example, with the eye winking) Then, the intensity leveldetected during a wink can be compared to the stored eye-closedintensity levels. In this way, the gaze direction can be determined bythe light data received during the wink in addition to the light datareceived before and after the wink.

In some embodiments, the system can determine a gaze direction withoutreferring to a list of representative data. For example, if the winkgesture occurs while the eye is looking forward, the difference betweenthe light-intensity level before the wink gesture and thelight-intensity level during the wink gesture can be much larger than ifthe user were looking either up or down. Therefore, the system candetermine a first light-intensity level associated with an eye-openstate and a second light-intensity level associated with an eye-closedstate. Further, the system can determine that the difference in lightintensity is greater than a non-zero threshold difference and, based onthis determination, determine that the gaze direction is an intermediatevertical direction (i.e., between an upward and a downward direction).Similarly, the system can determine that the gaze direction is one of anupward direction and a downward direction, in response to determiningthat the difference in light intensity is not greater than a non-zerothreshold. Similar procedures can be used for comparing the intensityduring a wink to the intensity after the wink.

c. Capturing Image Data

With reference to FIG. 4, at block 404, the method 400 involves causingan image capture device to capture image data, in response to detectingthe wink gesture at the HMD. As used in this disclosure, the term “imagedata” can refer to various types of data; the meaning of the term “imagedata” can depend on the context in which the term is used. In somecontexts, the term “image data” can refer to a raw image file (or tomultiple raw image files). The raw image file can represent unprocessedor minimally processed data from an image sensor of a camera, such as adigital camera or an image scanner, among other types. Examples of rawimages files include camera image file format (CIFF) and digitalnegative (DNG). Note that this disclosure contemplates any othersuitable type of raw image file. In some contexts, the term “image data”can refer to data in a format that can be rasterized for use on adisplay; examples include RAW images, Portable Network Graphics (PNG)images, Joint-Photographic Experts Group (JPEG) compressed images,Bitmap (BMP) images, and Graphics Interchange Format (GIF) images, amongvarious other types. In some contexts, the term “image data” can referto data in a vector format, such as, for example, an eXtensible MarkupLanguage (XML) based file format; an example includes Scalable VectorGraphics (SVG), among other types. In some contexts, the term “imagedata” can refer to data that is in a graphics pipeline along a renderingdevice, such as a graphics processing unit (GPU) or a central processingunit (CPU), among others. In some contexts, the term “image data” canrefer to data that is stored in a display's video memory (such as, forexample, random access memory (RAM)) or in graphics card. In somecontexts, the term “image data” can refer to data that includeslight-field information, such as, for example, four-dimensional (4D)light-field information. In this example, the data can represent rawdata that is captured by, for example, a plenoptic camera (sometimestermed a “light-field camera”), or the data can represent a processedversion of such raw data. Note that the term “image data” can encompassvarious types of data, can be of various file formats, and can be storedto various mediums, whether those types of data, file formats, andmediums are known or have yet to be developed.

The image data can be, but need not be, data that was captured by acamera. Accordingly, the image capture device can be, but need not, be acamera. As an example, the image data can represent a still image of analready captured video, whether the still image is in the same fileformat as the video or in a different file format from the video. Inthis example, the image capture device includes any combination of thehardware, firmware, and software that is used to generate the stillimage from the frame of the video. Of course, in this example, the imagedata can represent multiple still images of the video. As anotherexample, the image data can represent a screenshot of a display. Theseexamples are illustrative only; image data can be captured in variousother ways.

In this disclosure, when an action is said to be performed in connectionwith captured image data, the action can be performed directly on theimage data itself, can be performed on a processed version of the imagedata, can be performed on different image data entirely, or can beperformed on a combination of these. The different image data can be aduplicate of the captured image data, a processed version of thecaptured image data, or a portion of the captured image data. Forexample, in some contexts, when image data is said to be sent to arecipient, the image data itself can be, but need not, be sent to therecipient; instead, different image data (a duplicate, a processedversion, a portion, or the like) can be sent to the recipient.

Also, in some contexts, this disclosure refers to “image data” as an“image” or as “images” for ease of explanation.

In some implementations, the image data can represent a point-of-viewimage. As used in this disclosure, a “point-of-view image” can be animage portraying the perspective of an actual user or it can represent aperspective that a user would have if the device were being worn. Adevice can capture the point-of-view image or send an image-captureinstruction to cause an image-capturing system to capture thepoint-of-view image. In some cases, the device can refrain fromcapturing a point-of-view image in response to detecting a wink withcertain characteristics. For example, a device can refrain fromcapturing image data in response to detecting a wink that lasts for arelatively short threshold duration.

A point-of-view image can be captured by one or more outward-facingvideo cameras, still-picture cameras, light sensor arrays, or otherimage-generating systems or devices. The image-capturing device can befixed in position and direction, or can be movable on the device. Insome cases, the image-capturing device can be movable by the processingcomponents of the HMD. In this way, a device can change the directionand position of an image-capturing device automatically. In otherarrangements, instead of physically moving a narrow-angle device to facea different direction, the image-capturing device can capture awide-angle image and then crop the image to store a particular directionin the captured image.

In some arrangements, the instruction to capture the image (whetherinstructing integral, local, or remote image-capturing systems) cancontain specific instructions on how to perform the image capture. Forexample, the instructions can direct the image-capturing device to focusthe capture on a point or part of the device's field of view. Inparticular, the image-capturing device can be configured to have animaging field of view, which includes at least a portion of the field ofview of the device's user. As such, locations within the field of viewof the wearer can be mapped to locations within the imaging field ofview of the image-capturing device. In this way, the gaze direction canbe directed toward a point or area of interest within the field of viewof the user. The point or area of interest within the field of view ofthe user can then be mapped to a corresponding location within theimaging field of view of the image-capturing device. Accordingly, theimage capture instruction can be generated to indicate the correspondinglocation within the imaging field of view on which the image-capturingdevice should focus, thereby focusing on areas in the direction alongwhich the user is looking.

In some embodiments, a device can capture image data in response to oneor more winks during which the eye is oriented along a certaindirection. In particular, the device can make a determination whetherthe gaze direction is one of a predetermined set of directions and thencapture image data in response to determining that the gaze direction isone of the predetermined directions. For example, the device can store arange of directions that are “on screen” and a range of directions thatare “off screen”. When the user is looking relatively forward (i.e.,towards an intermediate vertical direction, as shown in FIG. 2) whenwinking, the device can recognize the wink gesture as on-screen winkingand responsively capture a point-of-view image. In contrast, when theuser is looking upward or downward while winking, the device canrecognize the wink gesture as off-screen winking and responsivelyrefrain from capturing the point-of-view image.

In addition to capturing image data, a device can perform other actionsin response to detecting the wink gesture. For example, the device canpresent a notification of the image capture. As another example, thedevice can responsively transmit the captured image data to otherdevices or servers.

d. Detecting a Secondary Gesture at an HMD

As discussed above, the method 400 involves detecting a wink gesture atan HMD. Some implementations of the method 400 can also involvedetecting a secondary gesture at the HMD. In addition, someimplementations of the method 400 can involve causing an image capturedevice to capture image data, in response to detecting the secondarygesture at the HMD.

i. Secondary Gesture

The term “secondary gesture,” as used in this disclosure, generallyrefers to an action or combination of actions that a wearer of an HMD(or simply “wearer”) performs in addition to a wink gesture.Accordingly, when a wink gesture is discussed in connection with thesecondary gesture, the wink gesture is sometimes referred to as a“primary gesture” for ease of explanation. Depending on the desiredimplementation, a secondary gesture can encompass an intentional gesture(such as, for example, a change in the wearer's gaze direction), anunintentional gesture (such as, for example, a reflexive blink), orboth. In addition, the term “secondary gesture” can also encompassinactivity, depending on the context in which the term is used.

The meaning of the term “secondary gesture,” when used in connectionwith an HMD, can depend on the configuration of the HMD. Several HMDconfigurations serve as illustrative examples. A first illustrative HMDconfiguration can recognize a secondary gesture as an action or acombination of actions that is performed in connection with the HMD. Inthis HMD configuration, the action or combination of actions is said toserve as the secondary gesture, or in other words, result in a detectionof the secondary gesture. A second illustrative HMD configuration canrecognize a secondary gesture as inactivity with respect to the HMD. Inthis HMD configuration, the inactivity serves as the secondary gesture.A third illustrative HMD configuration can recognize a secondary gestureas a combination of inactivity with a suitable action or combination ofsuitable actions. In this HMD configuration, the combination ofinactivity with a suitable action or combination of suitable actionsserves as the secondary gesture.

When an HMD is worn, its configuration can permit an eye-related actionto serve as a secondary gesture. Several illustrative examples ofeye-related actions follow. As a first illustrative example, a squintgesture can serve as a secondary gesture. The squint gesture can takethe form of a squint gesture as discussed above. The squint gesture caninclude a squint of one or both of the wearer's eyes. In addition, thesquint gesture can include one squint or multiple squints. As a secondillustrative example, a blink gesture can serve as a secondary gesture.The blink gesture can take the form of a blink gesture as discussedabove. A blink gesture typically includes a blink of both of thewearer's eyes, but the blink gesture can also be a blink of just one ofthe wearer's eyes. In addition, the blink gesture can include a singleblink or multiple blinks; the multiple blinks can include one blink ofeach of the wearer's eyes or multiple blinks of the same eye. As a thirdillustrative example, a change in gaze direction can serve as asecondary gesture. The change in gaze direction can be as discussedabove. As a fourth illustrative example, the secondary gesture can takethe form of another wink gesture—in other words, a wink gesture inaddition to the primary gesture.

Some examples in this disclosure discuss situations in which the wearerperforms the secondary gesture while performing the wink gesture. Inthese examples, the wearer's winking eye is closed when the wearerperforms the secondary gesture; therefore, in these examples, aneye-related action that serves as a secondary gesture is performed usingthe wearer's other eye.

When an HMD is worn, its configuration can permit an action other thanan eye-related action to serve as a secondary gesture. Severalillustrative examples of these actions follow. As a first illustrativeexample, a threshold movement of an HMD can serve as a secondarygesture. To this end, the HMD can include a sensor system that is ableto detect movements of the HMD. The sensor system can include devicessuch an accelerometer, a gyroscope, a proximity sensor, or similardevices. Of course, other devices and configurations can be used todetect movements of the HMD. Note that the threshold movement typicallyoccurs when a wearer is wearing the HMD. For instance, the HMD can beconfigured so that a head nod serves as the threshold movement and,therefore, as the secondary gesture. As a further refinement, a head nodin a first direction—for example, an upward head nod—can serve as afirst threshold movement and, therefore, as a first type of secondarygesture; a head nod in a second direction—for example, a downward headnod—can serve as a second threshold movement and, therefore, as a secondtype of secondary gesture. In this example, the first and the secondsecondary gesture can correspond to different functions of the HMD. Forinstance, the first secondary gesture (i.e., the upward head nod) cancause the HMD to remove a displayed image from display, whereas thesecond secondary gesture (i.e., a downward head nod) can cause the HMDto save a displayed image.

Depending on the HMD's configuration, the threshold movement can occuras the wearer is removing the HMD or when the HMD is not worn. As asecond illustrative example, a voice command can serve as a secondarygesture. To this end, an HMD can be equipped with a voice-commandinterface. The voice-command interface can enable the wearer or anotherperson to issue spoken commands to the HMD. These spoken commands aretermed “voice commands” for ease of explanation. As a third illustrativeexample, a gesture at a finger-operable device can trigger a secondarygesture. To this end, an HMD can be equipped with a finger-operabledevice, such as, for example, the finger-operable touch pad 124discussed above in connection with FIGS. 1A and 1B. Note that anycombination of the actions discussed above can serve as a secondarygesture. In addition, the actions discussed above are illustrative only,so other actions can also serve as secondary gestures.

When an HMD is worn, its configuration can permit inactivity to serve asa secondary gesture. In particular, inactivity by itself can serve as asecondary gesture or the inactivity in combination with one or moreactions can serve as the secondary gesture. As an illustrative example,the inactivity can represent a completion of a period in which a wearerof an HMD does not perform a suitable action, such as, for example, oneof the actions discussed above. As another illustrative example, asuitable action, such as a head nod, can start a threshold period. Upondetermining that the threshold period has ended without suitableactivity, the HMD can determine that the secondary gesture has occurred.

Of course, when an HMD is worn, its configuration can permit anycombination of eye-related actions, other actions, and inactivity toserve as a secondary gesture.

ii. Secondary Gesture Detection System

An HMD can include a secondary gesture detection system for detectingsecondary gestures. The term “secondary gesture detection system,” asused in this disclosure, generally refers to any combination of asystem, device, or set of instructions that is configured to detect oneor more secondary gestures at an HMD. The meaning of the term “secondarygesture detection system” can depend on an HMD's configuration. Severalexamples are illustrative.

As a first illustrative example, assume that in an HMD, a wink gestureserves as a secondary gesture. In this HMD, a wink detection system,such as the wink detection system discussed above in connection withFIGS. 1A-1B, can also serve as the secondary gesture detection system.

As a second illustrative example, assume that in an HMD, a voice commandserves as a secondary gesture. In this HMD, a system for detecting voicecommands can serve as the secondary gesture detection system. Note thatthe system for detecting the voice commands can include software,hardware, or both, depending on the desired implementations of the HMD.

As a third illustrative example, assume that in an HMD, a gesture at afinger-operable device serves as a secondary gesture. In this HMD, thefinger-operable device, either by itself or in combination with otherhardware or software, can serve as the secondary gesture detectionsystem. For instance, the finger-operable touch pad 124 discussed abovein connection with FIGS. 1A-1B can serve as the secondary gesturedetection system.

The secondary gesture detection system can also be configured to detectmultiple types of secondary gestures. As an illustrative example, assumethat in an HMD, a wink gesture and a blink gesture each serve as asecondary gesture. In this HMD, a system that is configured to detect ablink gesture and a wink gesture, such as the system discussed above,can serve as the secondary gesture detection system.

Note that these examples are illustrative only. The secondary gesturedetection system can include various systems, devices, or sets ofinstructions, whether these are currently known or have yet to bedeveloped. In addition, the secondary gesture system can includehardware, software, or a combination of hardware and software.

iii. Image Capture Device

As mentioned above, the method 400, at block 404, can involve causing animage capture device to capture image data, in response to detecting asecondary gesture at an HMD. The image capture device can be a camera,another photographic device, or any combination of hardware, firmware,and software that is configured to capture image data.

The image capture device can be disposed at the HMD or apart from theHMD. As an illustrative example, the image capture device can be aforward-facing camera, such as the video camera 120 that is discussedabove in connection with FIGS. 1A and 1B. As another illustrativeexample, the image capture device can be a camera that is separate fromthe HMD and in communication with the HMD with the use of a wired orwireless connection. Note that any suitable camera or combination ofcameras can serve as the image capture device. Examples of suitablecameras include a digital camera, a video camera, a pinhole camera, arangefinder camera, a plenoptic camera, a single-lens reflex camera, orcombinations of these. These examples are merely illustrative; othertypes of cameras can be used.

e. Triggering HMD Functionality Based on One or More Gestures

In general, a wink gesture or a combination of gestures that includes awink gesture can trigger HMD functionality.

i. Examples of HMD Functionality

The term “HMD functionality,” as used in this disclosure, generallyrefers to any function or combination of functions that is performedwith the use of an HMD. Examples of HMD functionality include thefollowing: (1) activating, deactivating, or modifying an interface ofthe HMD; (2) capturing image data using a camera of the HMD, or a camerathat is separate from and in communication with the HMD; (3) detecting aface based on an analysis of image data; (4) detecting an object basedon analysis of image data; (5) recording a video using a camera of theHMD, or a camera that is separate from and in communication with theHMD; (6) displaying a video using a display of the HMD, or a displaythat is separate from and in communication with the HMD; (7) sendingimage data in an e-mail; (8) sending image data to a social network; (9)sending information to another device, such as a mobile phone or anotherHMD; (10) activating or de-activating the HMD itself; (11) activating ordeactivating a display of the HMD, or a display that is separate fromand in communication with the HMD; (12) modifying information providedin a display of the HMD, or a display that is separate from and incommunication with the HMD; (13) using the HMD to activate or deactivatean external device; and (14) causing a display device to display animage, based on the image data; and (15) any combination of these. Theseexamples are illustrative only; there are numerous other examples of HMDfunctionality. This disclosure contemplates HMD functionality that isnot expressly discussed.

ii. Timing of Combined Gestures

In general, to trigger HMD functionality, the wink gesture and thesecondary gesture can occur one after the other or simultaneously. Theorder of their occurrence can depend on the desired implementation ofthe HMD. Several HMD implementations serve as illustrative examples. Afirst HMD implementation can allow a wink gesture followed by asecondary gesture to trigger HMD functionality. As an example, assumethat a wearer winks and then nods his head. Also assume that the winkrepresents the wink gesture and that the head nod represents thesecondary gesture. Upon detecting the wink gesture and the secondarygesture in this particular order, the HMD can carry out the HMDfunctionality.

A second HMD implementation can allow a secondary gesture followed by awink gesture to trigger HMD functionality. As an example, assume that awearer says, “take a photo” and then winks twice. Also assume that thevoice command “take a photo” represents the secondary gesture and thatthe two winks represent the wink gesture. Upon detecting the winkgesture and the secondary gesture in this particular order, the HMD cancarry out the HMD functionality.

A third HMD implementation can allow a wink gesture and a simultaneouslyoccurring secondary gesture to trigger HMD functionality. As an example,assume that a wearer begins a wink with one eye and that, while holdingthe wink, the wearer changes the gaze direction of his other eye. Assumethat after changing the gaze direction, the wearer releases the wink.Also assume that the wink represents the wink gesture and that thechange of gaze direction represents the secondary gesture. Upondetecting that the secondary gesture occurs during the wink, the HMD cancarry out the HMD functionality. For example, the HMD can capture imagedata upon detecting that the change in gaze direction (the secondarygesture) occurs while the wearer is holding the wink (the wink gesture).Note that in this example, the wink gesture and the secondary gestureneed not start and end at the same time to trigger the HMDfunctionality; instead, it suffices that the wink gesture and thesecondary gesture temporally overlap. However, some HMD configurationscan require that the wink gesture and the secondary gesture start andend at the same time to trigger HMD functionality.

These HMD implementations can be combined in any way. In one combinationof these HMD implementations, a secondary gesture followed by a winkgesture can trigger a first HMD functionality, while a wink gesturefollowed by a secondary gesture can trigger a different, second HMDfunctionality. As an example, a secondary gesture followed by a winkgesture can trigger a voice-command interface, whereas a wink gesturefollowed by a secondary gesture can trigger a camera mode. In anothercombination of these HMD implementations, a secondary gesture followedby a wink gesture can trigger HMD functionality, and the wink gesturefollowed by the secondary gesture can trigger the same HMDfunctionality.

iii. Using a Wink Gesture in Combination with a Secondary Gesture toTrigger Multiple HMD Functions

A wink gesture and a secondary gesture can occur in combination totrigger multiple functions of an HMD. Some HMD configurations permit awink gesture to trigger a first HMD function and permit a secondarygesture to trigger a second HMD function. Several illustrative examplesfollow. As a first illustrative example, assume that a wearer winks andthen changes a gaze direction. Also assume that the wink represents thewink gesture and that the change of gaze direction represents thesecondary gesture. Upon detecting the wink gesture, the HMD can initiatea camera mode, in which a display of the HMD displays the real-timeimage stream captured by a lens of the camera. In this way, the displayserves as a viewfinder of the camera, and therefore, the real-time imagestream is referred to a “viewfinder image” for ease of explanation.Then, upon detecting the secondary gesture, the HMD can activate,deactivate, or modify a photographic function of the HMD. For instance,the HMD can activate a flash function of the camera, upon detecting thesecondary gesture.

As a second illustrative example, assume that the order of the winkgesture and the secondary gesture is reversed; that is, the wearerchanges the gaze direction (secondary gesture) and then winks (winkgesture). In this example, upon detecting the secondary gesture, the HMDcan initiate the camera mode. Then, upon detecting the wink gesture, theHMD can activate, deactivate, or modify the photographic function.

As a third illustrative example, assume that a wearer begins a wink withone eye and that, while holding the wink, the wearer issues the voicecommand “Turn off flash.” Assume that after issuing the voice command,the wearer releases the wink. Also assume that the wink represents thewink gesture and that the voice command represents the secondarygesture. In this example, when the wearer begins the wink, the HMDactivates a camera mode (as discussed above). Then, when the wearerissues the voice command, the HMD activates, deactivates, or modifies aphotographic function, such as, for example, the flash function of thecamera. Then, when the wearer releases the wink, the HMD captures imagedata. In this way, the wearer can start and hold a wink to enter acamera mode, perform secondary gestures to activate, deactivate, ormodify a photographic function (while continuing to hold the wink), andthen release the wink to capture image data.

As a fourth illustrative example, assume that a wearer winks and thennods his head. Also assume that the wink represents the wink gesture andthat the head nod represents the secondary gesture. In this example,when the wearer winks, the HMD can capture image data, for example, bycausing a camera of the HMD to capture an image. In someimplementations, a display, such as a display of the HMD, can display animage that corresponds to the image data. The image can be displayedautomatically—in other words, without a need for user intervention.Then, when the wearer nods his head, the HMD can perform a predeterminedfunction, such as, for example, causing the display to remove the imagefrom display, storing the image, sending the image to a social network,or sending the image in an e-mail, among other functions. In this way,the HMD can display an image in response to a wink and can then enable asecondary gesture to interact with the displayed image.

In general, in response to detecting a wink gesture, a secondarygesture, or both, an HMD can activate, deactivate, or modify anyphotographic function. Examples of photographic functions include thefollowing: a zoom function, a flash function, an auto-focus function, amenu in a camera mode, a photo filter, an image size, a burstphotography function, a high dynamic range function, and any combinationof these. These examples are illustrative only; this disclosurecontemplates various other types of photographic functions, regardlesswhether those functions are now known or later developed.

iv. Using an Additional Gesture in Combination with a Wink Gesture andSecondary Gesture to Trigger Multiple HMD Functions

An additional gesture can occur in combination with the wink gesture andsecondary gesture to trigger multiple HMD functions. The additionalgesture can be another wink gesture or any other suitable gesture. As anillustrative example, assume that a wearer winks, then changes a gazedirection, and then winks again. In this example, the first wink (thewink gesture) triggers a camera mode, the change in gaze direction (thesecondary gesture) causes the HMD to change a photographic function, andthe second wink (the additional gesture) causes the HMD to capture imagedata.

The discussion above in relation to wink gestures, secondary gesture,and additional gestures is merely illustrative. In general, thisdisclosure contemplates any situation in which a wink gesture or acombination of gestures that includes a wink gesture can trigger HMDfunctionality. When a combination of gestures is involved, the gesturesin the combination can be performed in any order to trigger HMDfunctionality. In addition, the triggered HMD functionality canconstitute any function or combination of functions that can beperformed in connection with an HMD. The HMD functionality is notlimited to those functions discussed above. Nor is the HMD functionalitylimited to functions that physically occur at the HMD itself. The HMDfunctionality can include functions that occur elsewhere with the use ofthe HMD.

f. Triggering HMD Functionality Based on a Context in which an HMDOperates

As discussed above, a wink gesture or a combination that includes a winkgesture can trigger HMD functionality. In some HMD implementations, thetriggered HMD functionality need not be the same under allcircumstances; rather, the HMD functionality to be triggered can dependon the context in which the HMD operates. Several illustrative examplesfollow.

As an illustrative example, the HMD functionality to be triggered candepend on the HMD's location. Assume that an HMD detects a wink gesturethat is suitable to trigger HMD functionality. Also assume that the HMDis able to utilize data from a sensor system, such as a GPS system, todetermine the HMD's location. The HMD can use the determined location toidentify a setting in which the HMD is located. Based on the identifiedsetting, the HMD can trigger appropriate HMD functionality. If a wearerof an HMD were to wink at a beach, for example, then the HMD wouldinitiate a camera mode that is optimized to take pictures in an outdoorsetting, such as a beach setting. But if the HMD wearer were to wink ina house, for example, then the HMD would responsively initiate a cameramode that is optimized to take pictures in an indoor setting, such asthe interior of the house.

As another illustrative example, the HMD functionality to be triggeredcan depend on the time, by itself or in combination with a determinedlocation. Assume that an HMD detects a wink gesture that is suitable totrigger HMD functionality. Also assume that the HMD is able to determinethe local time. Based on the determined time, the HMD can triggerappropriate HMD functionality. If a wearer of an HMD were to wink whilestanding outside at 9:30 PM, for example, then the HMD would initiate acamera mode with the flash function on, due to the relatively low levelof ambient light near the HMD at that time. But if the HMD wearer wereto wink while standing outside at 2:30 PM, for example, then the HMDwould initiate the camera mode with the flash off, due to the relativelyhigh level of ambient light near the HMD at that time. Note that an HMDneed not detect time to determine the level of ambient light near theHMD. Instead, the HMD could use an ambient light sensor to detect theambient light level near the HMD.

As another illustrative example, the HMD functionality to be triggeredcan depend on the identity of the HMD's wearer. Assume that the HMDdetects a wink gesture that is suitable to trigger HMD functionality.Also assume that the HMD is able to determine the identity of itswearer. Assume further that the HMD has stored preferences for usersAlice and Bob. Thus, if Bob were to wear the HMD while winking, forexample, then the HMD would capture image data and store the image datato the HMD. But if Alice were to wear the HMD while winking, forexample, then the HMD would capture image data and send the image datato a social network.

This disclosure is not limited to these illustrative examples; rather,this disclosure contemplates any other suitable technique for triggeringHMD functionality, based on a context in which the HMD operates.

g. Winking to Start or End a Photographic Process

As discussed above, the method 400, at block 402, involves detecting awink gesture at an HMD. Some implementations of the method 400 can alsoinvolve causing a photographic process to commence, in response todetecting the wink gesture at the HMD. Several illustrative examplesfollow. As an illustrative example, an HMD can commence a time-lapsephotography process, in response to detecting the wink gesture at theHMD. For instance, the time-lapse photography process can involvecapturing multiple sets of image data at spaced time intervals. Asanother illustrative example, an HMD can start a process of displaying avideo on a display device of the HMD, in response to detecting the winkgesture at the HMD. As another illustrative example, the HMD can start aprocess or recording a video using a camera of the HMD, in response todetecting the wink gesture at the HMD.

In addition, some implementations of the method 400 can involve causinga photographic process to end, in response to detecting a wink gestureat the HMD. As an illustrative example, assume that a wearer of an HMDwinks, with the wink representing a first wink gesture. Upon detectingthe first wink gesture, the HMD can begin a time-lapse photographyprocess. Assume that while the time-lapse photography process is active,the wearer winks again, with that wink representing a second winkgesture. Upon detecting the second wink gesture, the HMD can end thetime-lapse photography process.

Some implementations of the method 400 can involve detecting a secondwink gesture, which can be a wink gesture that occurs before the winkgesture detected at block 402. These implementations can also involveactivating the photographic function at the HMD, in response todetecting the second wink gesture at the HMD. As an illustrativeexample, assume that an HMD wearer winks, with the wink representing afirst wink gesture. Upon detecting the first wink gesture, the HMD canbegin recording a video using a camera of the HMD. Assume that while theHMD is recording the video, the HMD wearer winks again, with that winkrepresenting a second wink gesture. Upon detecting the second winkgesture, the HMD can capture the present frame of the video as imagedata. Of course, the HMD can also capture multiple frames of the videoas image data.

As discussed above, the method 400, at block 404, involves causing animage capture device to capture image data, in response to detecting awink gesture at the HMD. In some implementations that involve causing aphotographic process to commence, block 404 of the method 400 caninvolve capturing image data that results from the photographic process.Several illustrative examples follow. As a first illustrative example,assume that the photographic process is a time-lapse photographyprocess. In this example, block 404 can involve capturing one set ofimage data or multiple sets of image data that result from thetime-lapse photography process. As a second illustrative example, assumethat the photographic process involves displaying a video on a displaydevice. In this example, block 404 can involve capturing one or moreframes of the video. As a third illustrative example, assume that thephotographic process involves recording a video using a camera. In thisexample, block 404 can similarly involve capturing one or more frames ofthe video.

h. Using Image Data as Input to a Function of an HMD

As discussed above, the method 400, at block 404, involves causing animage capture device to capture image data, in response to detecting thewink gesture at the HMD. Some implementations of the method 400 caninvolve using the captured image data as input to a function of the HMD.

The image data can be used as input to any function of the HMD. Examplesof HMD functions include the following: (1) activating, deactivating, ormodifying an interface, such as a voice command interface of the HMD;(2) displaying an image corresponding to the image data at a display,such as a display of the HMD or a separate display; (3) detecting a facebased on the image data; (4) detecting an object based on the imagedata; (5) recording a video using a camera of the HMD; (6) displaying avideo using a display of the HMD; (7) sending the image data in ane-mail; (8) sending the image data to a social network; (9) sendinginformation to another device, such as a mobile phone or another HMD;(10) activating or de-activating the HMD itself; (11) activating ordeactivating a display of an HMD; (12) modifying information provided ina display of an HMD; (13) using the HMD to activate, deactivate, ormodify an external device, such as an external camera or display; and(14) any combination of these. These examples are merely illustrative;there are numerous other types of HMD functions. This disclosurecontemplates HMD functions that are not expressly discussed.

The following examples serve to illustrate how image data can be used asinput to a function of an HMD. For the purpose of several illustrativeexamples, assume that the HMD is able to cause a camera to capture imagedata. The camera can be part of the HMD or can be a separate camera thatthe HMD is able to control in a wired or wireless fashion. Also assumethat the HMD is able to detect a person's face based on an analysis ofthe image data, such as, for example, by using a suitablefacial-detection algorithm. Upon detecting the face, the HMD can attemptto identify the person, based on the face. To this end, the HMD cananalyze the face with reference to information stored at the HMD or toinformation stored remotely. Upon identifying the person, the HMD cansend the image data to an account that is associated with the person.For example, the HMD can send the image data as an e-mail attachment tothe person's e-mail address. As another example, the HMD can send theimage data to the person's phone number using a multimedia messagingservice. As yet another example, the HMD can post the image data to theperson's social-network account or, in other words, send the image datato the social network in a way that the image data becomes associatedwith the person's social-network account.

4. Conclusion

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

We claim:
 1. A method comprising: detecting a wink gesture at ahead-mountable device (HMD); determining a context of the HMD when thewink gesture is detected; activating a photographic function at the HMDbased on the determined context; and causing an image capture device tocapture image data, in response to detecting the wink gesture at theHMD.
 2. The method of claim 1, wherein the context is selected from thegroup consisting of: a local time, an ambient light level, and alocation of the HMD.
 3. The method of claim 1, wherein the photographicfunction is selected from the group consisting of: a zoom function, aflash function, an autofocus function, a burst photography function, ahigh dynamic range function, and any combination thereof.
 4. The methodof claim 1, further comprising detecting a secondary gesture at the HMD,wherein the causing the image capture device to capture the image datais performed further in response to the detecting the secondary gestureat the HMD.
 5. The method of claim 4, further comprising activating aphotographic function at the HMD, in response to the detecting thesecondary gesture at the HMD.
 6. The method of claim 4, wherein thesecondary gesture is selected from the group consisting of: a squintgesture, a blink gesture, a movement of the HMD, an eye movement, avoice command, a head nod, a gesture at a finger-operable devicedisposed at the HMD, an inaction during a threshold period, and anycombination thereof.
 7. The method of claim 1, further comprisingcausing a display device to display a video, wherein the causing theimage capture device to capture the image data comprises capturing atleast a part of the video.
 8. The method of claim 1, further comprisingcommencing a photographic process, in response to the detecting the winkgesture at the HMD.
 9. The method of claim 8, wherein the photographicprocess is selected from the group consisting of: a time-lapsephotographic process, a process of recording a video, a process ofdisplaying a video, and any combination thereof.
 10. The method of claim8, wherein the image data is captured based on data that results fromthe photographic process.
 11. The method of claim 1, further comprisingusing the image data as an input to a function of the HMD.
 12. Themethod of claim 11, wherein the function of the HMD is selected from thegroup consisting of: performing an analysis of the image data, sendingthe image data to a social network, sending the image data in a message,and any combination thereof.
 13. The method of claim 11, wherein theusing the image data as the input to the function of the HMD isperformed without displaying an image corresponding to the image data.14. The method of claim 1, further comprising: detecting a face of aperson, based on an analysis of the image data; identifying the face;and sending the image data to an account that is associated with theperson.
 15. A non-transitory computer-readable medium having storedtherein instructions that, upon execution by a computing device, causethe computing device to perform functions comprising: detecting a winkgesture at a head-mountable device (HMD); determining a context of theHMD when the wink gesture is detected; activating a photographicfunction at the HMD based on the determined context; and causing animage capture device to capture image data, in response to detecting thewink gesture at the HMD.
 16. The non-transitory computer-readable mediumof claim 15, wherein the context is selected from the group consistingof: a local time, an ambient light level, and a location of the HMD. 17.The non-transitory computer-readable medium of claim 15, wherein thephotographic function is selected from the group consisting of: a zoomfunction, a flash function, an autofocus function, a burst photographyfunction, a high dynamic range function, and any combination thereof.18. A system of a head-mountable device (HMD), the system comprising: animage capture device that is connected to the HMD, wherein when the HMDis worn, the image capture device is configured to capture image data; awink-detection system that is connected to the HMD, wherein when the HMDis worn, the wink-detection system is configured to detect a winkgesture at the HMD; and a computer-readable medium having stored thereinprogram instructions that, upon execution by a computing device, causethe computing device to perform functions comprising: determining acontext of the HMD when the wink gesture is detected; activating aphotographic function at the HMD based on the determined context; andcausing the image capture device to capture the image data, in responseto the wink-detection system detecting the wink gesture at the HMD. 19.The system of claim 18, wherein the context is selected from the groupconsisting of: a local time, an ambient light level, and a location ofthe HMD.
 20. The system of claim 18, wherein the photographic functionis selected from the group consisting of: a zoom function, a flashfunction, an autofocus function, a burst photography function, a highdynamic range function, and any combination thereof.